from sklearn_benchmarks.reporting.hp_match import HPMatchReporting

Benchmarking prediction: scikit-learn vs. ONNX Runtime

reporting = HPMatchReporting("onnx", config="config.yml")
reporting.run()

KNeighborsClassifier_brute_force

onnx (1.9.0) vs. scikit-learn (1.0.dev0)

All estimators share the following hyperparameters: algorithm=brute.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration	std_duration	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.774	0.025	0.0	0.017	-1	5	0.705	2.783	0.037	6.027	0.080	See	See
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.774	0.025	0.0	0.017	-1	5	0.705	0.024	0.003	700.990	80.179	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.296	0.008	0.0	0.296	-1	5	1.000	2.783	0.037	0.106	0.003	See	See
3	KNeighborsClassifier_brute_force	predict	100000	1	100	0.296	0.008	0.0	0.296	-1	5	1.000	0.024	0.003	12.386	1.452	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.483	0.015	0.0	0.017	1	1	0.642	1.121	0.002	15.589	0.026	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.483	0.015	0.0	0.017	1	1	0.642	0.019	0.000	913.377	23.235	See	See
6	KNeighborsClassifier_brute_force	predict	100000	1	100	0.302	0.011	0.0	0.302	1	1	1.000	1.121	0.002	0.270	0.010	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1	100	0.302	0.011	0.0	0.302	1	1	1.000	0.019	0.000	15.798	0.692	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.221	0.023	0.0	0.017	1	100	0.815	2.152	0.007	8.002	0.027	See	See
9	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.221	0.023	0.0	0.017	1	100	0.815	0.020	0.001	851.579	21.205	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1	100	0.291	0.005	0.0	0.291	1	100	1.000	2.152	0.007	0.135	0.003	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.291	0.005	0.0	0.291	1	100	1.000	0.020	0.001	14.381	0.449	See	See
12	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.086	0.034	0.0	0.017	-1	1	0.642	1.762	0.049	9.695	0.270	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.086	0.034	0.0	0.017	-1	1	0.642	0.023	0.002	757.263	58.129	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.299	0.005	0.0	0.299	-1	1	1.000	1.762	0.049	0.170	0.006	See	See
15	KNeighborsClassifier_brute_force	predict	100000	1	100	0.299	0.005	0.0	0.299	-1	1	1.000	0.023	0.002	13.262	1.042	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.808	0.034	0.0	0.017	-1	100	0.815	2.813	0.036	5.976	0.078	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.808	0.034	0.0	0.017	-1	100	0.815	0.023	0.003	720.665	88.837	See	See
18	KNeighborsClassifier_brute_force	predict	100000	1	100	0.298	0.004	0.0	0.298	-1	100	1.000	2.813	0.036	0.106	0.002	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1	100	0.298	0.004	0.0	0.298	-1	100	1.000	0.023	0.003	12.776	1.585	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.771	0.016	0.0	0.017	1	5	0.705	2.148	0.005	7.806	0.019	See	See
21	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.771	0.016	0.0	0.017	1	5	0.705	0.020	0.000	829.202	17.011	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1	100	0.302	0.004	0.0	0.302	1	5	1.000	2.148	0.005	0.141	0.002	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	100	0.302	0.004	0.0	0.302	1	5	1.000	0.020	0.000	14.939	0.357	See	See
24	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.532	0.020	0.0	0.004	-1	5	0.873	2.600	0.026	1.359	0.016	See	See
25	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.532	0.020	0.0	0.004	-1	5	0.873	0.008	0.002	427.423	99.792	See	See
26	KNeighborsClassifier_brute_force	predict	100000	1	2	0.238	0.004	0.0	0.238	-1	5	1.000	2.600	0.026	0.091	0.002	See	See
27	KNeighborsClassifier_brute_force	predict	100000	1	2	0.238	0.004	0.0	0.238	-1	5	1.000	0.008	0.002	28.749	6.730	See	See
28	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.536	0.009	0.0	0.004	1	1	0.840	1.017	0.002	3.476	0.012	See	See
29	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.536	0.009	0.0	0.004	1	1	0.840	0.002	0.000	2110.145	235.383	See	See
30	KNeighborsClassifier_brute_force	predict	100000	1	2	0.236	0.004	0.0	0.236	1	1	1.000	1.017	0.002	0.232	0.004	See	See
31	KNeighborsClassifier_brute_force	predict	100000	1	2	0.236	0.004	0.0	0.236	1	1	1.000	0.002	0.000	140.832	15.916	See	See
32	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.608	0.052	0.0	0.004	1	100	0.883	1.998	0.002	1.806	0.026	See	See
33	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.608	0.052	0.0	0.004	1	100	0.883	0.003	0.000	1325.711	101.955	See	See
34	KNeighborsClassifier_brute_force	predict	100000	1	2	0.237	0.004	0.0	0.237	1	100	1.000	1.998	0.002	0.119	0.002	See	See
35	KNeighborsClassifier_brute_force	predict	100000	1	2	0.237	0.004	0.0	0.237	1	100	1.000	0.003	0.000	87.036	6.752	See	See
36	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.537	0.010	0.0	0.004	-1	1	0.840	1.640	0.031	2.156	0.041	See	See
37	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.537	0.010	0.0	0.004	-1	1	0.840	0.010	0.007	371.717	273.794	See	See
38	KNeighborsClassifier_brute_force	predict	100000	1	2	0.236	0.004	0.0	0.236	-1	1	1.000	1.640	0.031	0.144	0.004	See	See
39	KNeighborsClassifier_brute_force	predict	100000	1	2	0.236	0.004	0.0	0.236	-1	1	1.000	0.010	0.007	24.848	18.306	See	See
40	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.585	0.016	0.0	0.004	-1	100	0.883	2.605	0.030	1.376	0.017	See	See
41	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.585	0.016	0.0	0.004	-1	100	0.883	0.009	0.002	385.576	66.978	See	See
42	KNeighborsClassifier_brute_force	predict	100000	1	2	0.236	0.004	0.0	0.236	-1	100	1.000	2.605	0.030	0.091	0.002	See	See
43	KNeighborsClassifier_brute_force	predict	100000	1	2	0.236	0.004	0.0	0.236	-1	100	1.000	0.009	0.002	25.409	4.431	See	See
44	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.546	0.008	0.0	0.004	1	5	0.873	1.990	0.003	1.781	0.005	See	See
45	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.546	0.008	0.0	0.004	1	5	0.873	0.003	0.000	1311.851	87.503	See	See
46	KNeighborsClassifier_brute_force	predict	100000	1	2	0.238	0.004	0.0	0.238	1	5	1.000	1.990	0.003	0.120	0.002	See	See
47	KNeighborsClassifier_brute_force	predict	100000	1	2	0.238	0.004	0.0	0.238	1	5	1.000	0.003	0.000	88.199	6.046	See	See

KNeighborsClassifier_kd_tree

onnx (1.9.0) vs. scikit-learn (1.0.dev0)

All estimators share the following hyperparameters: algorithm=kd_tree.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration	std_duration	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	116.063	0.000	0.0	0.116	-1	1	0.962	0.467	0.006	248.697	3.089	See	See
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	116.063	0.000	0.0	0.116	-1	1	0.962	0.003	0.000	45381.401	8215.301	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.549	0.064	0.0	2.549	-1	1	1.000	0.467	0.006	5.462	0.153	See	See
3	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.549	0.064	0.0	2.549	-1	1	1.000	0.003	0.000	996.657	182.142	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	112.160	0.000	0.0	0.112	1	100	0.978	4.909	0.036	22.850	0.167	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	112.160	0.000	0.0	0.112	1	100	0.978	0.005	0.001	21821.268	5113.768	See	See
6	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.571	0.068	0.0	2.571	1	100	1.000	4.909	0.036	0.524	0.014	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.571	0.068	0.0	2.571	1	100	1.000	0.005	0.001	500.122	117.938	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	113.684	0.000	0.0	0.114	1	1	0.962	0.755	0.007	150.484	1.311	See	See
9	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	113.684	0.000	0.0	0.114	1	1	0.962	0.001	0.000	99406.980	22714.241	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.588	0.120	0.0	2.588	1	1	1.000	0.755	0.007	3.425	0.162	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.588	0.120	0.0	2.588	1	1	1.000	0.001	0.000	2262.633	527.559	See	See
12	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	115.280	0.000	0.0	0.115	-1	5	0.977	0.893	0.027	129.137	3.974	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	115.280	0.000	0.0	0.115	-1	5	0.977	0.004	0.001	32165.042	5916.950	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.570	0.073	0.0	2.570	-1	5	1.000	0.893	0.027	2.879	0.121	See	See
15	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.570	0.073	0.0	2.570	-1	5	1.000	0.004	0.001	717.006	133.477	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	115.522	0.000	0.0	0.116	1	5	0.977	1.487	0.031	77.690	1.642	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	115.522	0.000	0.0	0.116	1	5	0.977	0.002	0.000	62065.358	16202.314	See	See
18	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.558	0.076	0.0	2.558	1	5	1.000	1.487	0.031	1.720	0.063	See	See
19	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.558	0.076	0.0	2.558	1	5	1.000	0.002	0.000	1374.407	361.117	See	See
20	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	116.755	0.000	0.0	0.117	-1	100	0.978	2.896	0.038	40.316	0.526	See	See
21	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	116.755	0.000	0.0	0.117	-1	100	0.978	0.007	0.001	16874.778	3260.522	See	See
22	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.573	0.104	0.0	2.573	-1	100	1.000	2.896	0.038	0.888	0.038	See	See
23	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.573	0.104	0.0	2.573	-1	100	1.000	0.007	0.001	371.808	73.392	See	See
24	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.036	0.001	0.0	0.000	-1	1	0.877	0.022	0.001	1.669	0.077	See	See
25	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.036	0.001	0.0	0.000	-1	1	0.877	0.002	0.000	17.362	2.961	See	See
26	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	-1	1	1.000	0.022	0.001	0.233	0.009	See	See
27	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	-1	1	1.000	0.002	0.000	2.427	0.410	See	See
28	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.059	0.001	0.0	0.000	1	100	0.914	0.034	0.000	1.711	0.018	See	See
29	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.059	0.001	0.0	0.000	1	100	0.914	0.001	0.000	94.897	21.591	See	See
30	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	100	0.000	0.034	0.000	0.150	0.004	See	See
31	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	100	0.000	0.001	0.000	8.311	1.899	See	See
32	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.036	0.001	0.0	0.000	1	1	0.877	0.019	0.000	1.846	0.039	See	See
33	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.036	0.001	0.0	0.000	1	1	0.877	0.001	0.000	60.148	14.238	See	See
34	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	1	1.000	0.019	0.000	0.262	0.008	See	See
35	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	1	1.000	0.001	0.000	8.530	2.028	See	See
36	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.037	0.001	0.0	0.000	-1	5	0.903	0.023	0.001	1.596	0.049	See	See
37	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.037	0.001	0.0	0.000	-1	5	0.903	0.002	0.000	16.971	2.695	See	See
38	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	-1	5	1.000	0.023	0.001	0.223	0.008	See	See
39	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	-1	5	1.000	0.002	0.000	2.369	0.379	See	See
40	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.037	0.001	0.0	0.000	1	5	0.903	0.021	0.000	1.780	0.035	See	See
41	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.037	0.001	0.0	0.000	1	5	0.903	0.001	0.000	63.142	14.936	See	See
42	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	5	1.000	0.021	0.000	0.247	0.008	See	See
43	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	5	1.000	0.001	0.000	8.764	2.086	See	See
44	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.060	0.000	0.0	0.000	-1	100	0.914	0.036	0.003	1.665	0.120	See	See
45	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.060	0.000	0.0	0.000	-1	100	0.914	0.002	0.000	28.115	5.823	See	See
46	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	-1	100	0.000	0.036	0.003	0.143	0.011	See	See
47	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	-1	100	0.000	0.002	0.000	2.407	0.503	See	See

HistGradientBoostingClassifier_best

onnx (1.9.0) vs. scikit-learn (1.0.dev0)

All estimators share the following hyperparameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

	estimator	function	n_samples_train	n_samples	n_features	n_iter	mean_duration	std_duration	iteration_throughput	latency	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	predict	100000	1000	100	300	0.462	0.028	0.002	0.0	0.818	0.107	0.001	4.33	0.261	See	See